Engagement vs. attitude: Measurement invariance across item orderings

Casey Osorio-Duffoo2, Renata Garcia Prieto Palacios Roji1, Morgan Russell1, John Kulas1


1 Montclair State University
2 Harver

Introduction

The roots of employee (aka work; e.g., Schaufeli & Bakker, 2010) engagement research likely started with theoretical expansions of forms of employee participation and job involvement (e.g., Elloy, Everett, & Flynn, 1991). Like many other constructs within the I-O psychology field, our knowledge of engagement has been at least partially informed by its measurement and documented association with other work attitudes and behaviors. There are currently many engagement scales used for either academic and/or applied purposes (for example, the UWES, ISA, or Q12).

The current engagement survey is being developed within a larger study, with Schaufeli et el.’s (2002) vigor-dedication-absorption structure forming the substantive element structure: engagement is a mental state wherein employees: a) feel energized (Vigor), b) are enthusiastic about the content of their work and the things they do (Dedication), and c) are so immersed in their work activities that time seems compressed (Absorption). Additionally, these three components are each measured at three attitudinal levels: d) feeling (e.g., affect), e) thought (e.g., cognition), and f) action (e.g., behavior).

This study examines the impact of item ordering on factor structure using this measure that is intentionally saturated with two different structural components: 1) attitudinal, and 2) substantive.

Methods

There were two samples used for this study. The first was a snowball sample initiated by IO Psychology faculty and graduate students, which resulted in a total of 282 participants. There were four counterbalanced orderings of item presentations within this administration, as well as an additional 18 contextual items. In the second data collection initiative, Qualtrics panels were solicited. These US workforce representative adults responded to two counterbalanced orderings of the focal 18 items along with 2 additional contextual items. The respondents of the second sample included 343 who responded to attitudinally clustered items and 404 who responded to substantively clustered items.

Materials

Our 18-item engagement measure was crafted to be intentionally complex (each item is intended to load on two constructs). This complexity, however, derives from a crossing of the attitudinal components of affect, cognition, and behavior with the substantive engagement components of vigor, dedication, and absorption. The 6-point response scale is: Strongly Disagree, Disagree, Somewhat Disagree, Somewhat Agree, Agree, Strongly Agree. The item stems as well as their intended scale associations are presented in Table 1.

Results

Two omnibus confirmatory factor analyses (CFAs) were imposed on the data conforming to the two scale definitions reported in Table 1. Regardless of item ordering, across 1,025 respondents, both models showed fair fit (\(χ^2_{substantive}\)=995.34, df=132, RMSEA=0.09; \(χ^2_{attitudinal}\)=1,103.47, df=132, RMSEA=0.09). Additional fit indexes for the two models (as well as models run separately within each condition) are presented in Table 2.Figures 1 and 2 are visual representations of the omnibus CFA models. Table 3 presents unit-weighted scale scores across the three substantive and three attitudinal scales. Omnibus data α’s (across all conditions) were 0.81 (Absorption), 0.91 (Dedication), 0.78 (Vigor), 0.78 (Affect), 0.89 (Cognition), and 0.83 (Behavior).

The effects expected from the six types of item presentations did emerge in the anticipated direction in all conditions, but were quite moderate (for example, \(Δχ^2_{Cond1}\) = 9.55, ΔAIC_Cond1 = 10.53). MEasurement invariance was evaluated across a series of progressively greater restrictive conditions. Because our six conditions were realized across two qualitatively different sampling procedures, we applied our analyses of measurement invariance twice - once within our snowball sample which resulted in strong/scalar invariance for the attitudinal structure and mariganlly attaining scalar invariance for the substantive CFA (see Tables 4 and 5). Secondly, the follow-up Qualtrics panel respondents there was evidence of weak/metric invariance for the substantive CFA across the two Qualtrics item orderings (\(ΔΧ^2\) = 13.84, df = 15, p = .54), but a lack of invariance for the attitudinal model.

Table 1: Summary fit indices across item ordering conditions
Condition Model X Df RMSEA SRMR CFI TLI AIC
Condition 1 3-factor substantive 300.86 132 0.14 0.11 0.68 0.63 3,282.88
3-factor attitudinal 290.33 132 0.14 0.11 0.70 0.65 3,272.35
Condition 2 3-factor substantive 310.01 132 0.15 0.10 0.71 0.66 3,257.45
3-factor attitudinal 322.52 132 0.15 0.10 0.69 0.64 3,269.96
Condition 3 3-factor substantive 252.07 132 0.12 0.10 0.78 0.74 3,510.32
3-factor attitudinal 275.74 132 0.13 0.10 0.73 0.69 3,534.00
Condition 4 3-factor substantive 224.96 132 0.10 0.09 0.82 0.79 3,421.64
3-factor attitudinal 228.99 132 0.10 0.09 0.81 0.78 3,425.66
Condition 5 3-factor substantive 549.8 132 0.10 0.05 0.90 0.89 14,932.57
3-factor attitudinal 497.9 132 0.10 0.04 0.92 0.90 14,880.67
Condition 6 3-factor substantive 468.02 132 0.09 0.05 0.91 0.90 17,953.02
3-factor attitudinal 610.6 132 0.10 0.06 0.88 0.86 18,095.61
Overall 3-factor substantive 995.34 132 0.09 0.05 0.91 0.90 46,915.05
3-factor attitudinal 1,103.47 132 0.09 0.05 0.90 0.88 47,023.18
Omnibus Confirmatory Factor Analysis substantive structure.

Figure 1: Omnibus Confirmatory Factor Analysis substantive structure.

Omnibus Confirmatory Factor Analysis attitudinal structure.

Figure 2: Omnibus Confirmatory Factor Analysis attitudinal structure.

Discussion

When items are presented “together” in terms of conceptual cohesion (either grouped by attitudinal dimension or substantive dimension), the CFAs yielded slightly better fit indicators than when items were not presented in conceptually cohesive groupings. This was true across different levels of cohesive emphasis. There is evidence of at least partial invariance within both administrations. However, the idea of invariance has greater support within the initial four administration conditions than with the second sampling method.

The practical implication is that item ordering perhaps does not matter so much. We were in a unique circumstance regarding the opportunity to manipulate different meaningful groupings because our focal measure is intentionally complex. The item ordering literature does fairly consistently claim better reliability of assessment scores later in a sequence of similar items, and this was not evaluated in the current study, but could be the focus of future investigations.